Attention, Test Code is Low-quality!

In this paper, we describe the formatting guidelines for ACM SIG Proceedings. Software testing is an essential process during software development and maintenance for improving software quality. Test code, the artefact during software testing, has been widely used in many software quality assurance techniques. Traditionally, software quality assurance techniques, e.g., automatic bug repair, fault localization, test case prioritization, and mining API usage from test code are based on the hypothesis of a sound quality of the test code. However, via empirical study on four open source projects, we found that the quality of test code is quite low comparing with corresponding source code, and this might hurt the above software quality assurance techniques. In this paper, we studied more than 140,000 LOC(lines of code) test code from four large scale and widely used open source projects and found that it is common for test code to be unregulated and of low-quality in open source projects. First, the comment clone ratio, unreleased resource ratio and clone code ratio of test code is much higher than that of corresponding source code; second, bug-fixed coverage is down to 0. We have learned the following lessons: the quality of test code is quite low comparing with corresponding source code, and the low quality test code may misguide existing software quality assurance techniques.